Self-referenced tracking

ABSTRACT

A new tracking technique is essentially “sourceless” in that it can be used anywhere with no set-up, yet it enables a much wider range of virtual environment-style navigation and interaction techniques than does a simple head-orientation tracker. A sourceless head orientation tracker is combined with a head-worn tracking device that tracks a hand-mounted 3D beacon relative to the head. The system encourages use of intuitive interaction techniques which exploit proprioception.

CLAIM OF PRIORITY

This application claims priority under 35 USC §119(e) to provisionalU.S. Patent Application Ser. No. 60/178,797, filed on Jan. 28, 2000, theentire contents of which are hereby incorporated by reference.

BACKGROUND

This invention relates to self-referenced tracking.

Virtual reality (VR) systems require tracking of the orientation andposition of a user's head and hands with respect to a world coordinateframe in order to control view parameters for head mounted devices(HMDs) and allow manual interactions with the virtual world. Inlaboratory VR setups, this tracking has been achieved with a variety ofmechanical, acoustic, magnetic, and optical systems. These systemsrequire propagation of a signal between a fixed “source” and the tracked“sensor” and therefore limit the range of operation. They also require adegree of care in setting up the source or preparing the site thatreduces their utility for field use.

The emerging fields of wearable computing and augmented reality (AR)require tracking systems to be wearable and capable of operatingessentially immediately in arbitrary environments. “Sourceless”orientation trackers have been developed based on geomagnetic and/orinertial sensors. They allow enough control to look around the virtualenvironment and fly through it, but they don't enable the“reach-out-and-grab” interactions that make virtual environments sointuitive and which are needed to facilitate computer interaction.

SUMMARY

In one aspect, in general, the invention provides a new trackingtechnique that is essentially “sourceless” in that it can be usedanywhere with no set-up of a source, yet it enables a wider range ofvirtual environment-style navigation and interaction techniques thandoes a simple head-orientation tracker, including manual interactionwith virtual objects. The equipment can be produced at only slightlymore than the cost of a sourceless orientation tracker and can be usedby novice end users without any knowledge of tracking technology,because there is nothing to set up or configure.

In another aspect, in general, the invention features mounting a trackeron a user's head and using the tracker to track a position of alocalized feature associated with a limb of the user relative to theuser's head. The localized feature associated with the limb may includea hand-held object or a hand-mounted object or a point on a hand.

In another aspect, in general, the invention features mounting asourceless orientation tracker on a user's head and using a positiontracker to track a position of a first localized feature associated witha limb of the user relative to the user's head.

In another aspect, in general, the invention features tracking a pointon a hand-held object such as a pen or a point on a hand-mounted objectsuch as a ring or a point on a hand relative to a user's head.

In another aspect, in general, the invention features using a positiontracker to determine a distance between a first localized featureassociated with a user's limb and a second localized feature associatedwith the user's head.

In another aspect, in general, the invention features a position trackerwhich includes an acoustic position tracker, an electro-optical systemthat tracks LEDs, optical sensors or reflective marks, a videomachine-vision device, a magnetic tracker with a magnetic source held inthe hand and sensors integrated in the headset or vice versa, or a radiofrequency position locating device.

In another aspect, in general, the invention features a sourcelessorientation tracker including an inertial sensor, a tilt-sensor, or amagnetic compass sensor.

In another aspect, in general, the invention features mounting a displaydevice on the user's head and displaying a first object at a firstposition on the display device.

In another aspect, in general, the invention features changing theorientation of a display device, and, after changing the orientation ofthe display device, redisplaying the first object at a second positionon the display device based on the change in orientation.

In another aspect, in general, the invention features determining thesecond position for displaying the first object so as to make theposition of the first object appear to be fixed relative to a firstcoordinate reference frame, which frame does not rotate with the displaydevice during said changing of the orientation of the display device.

In another aspect, in general, the invention features displaying thefirst object in response to a signal from a computer.

In another aspect, in general, the invention features mounting awearable computer on the user's body, and displaying a first object inresponse to a signal from the wearable computer.

In another aspect, in general, the invention features displaying atleast a portion of a virtual environment, such as a fly-through virtualenvironment, or a virtual treadmill, on the display device.

In another aspect, in general, the invention features displaying agraphical user interface for a computer on the display device.

In another aspect, in general, the invention features first object beinga window, icon or menu in the graphical user interface.

In another aspect, in general, the invention features the first objectbeing a pointer for the graphical user interface.

In another aspect, in general, the invention features changing theposition of the first localized feature relative to the position trackerand, after changing the position of the first localized feature,redisplaying the first object at a second position on the display devicedetermined based on the change in the position of the first localizedfeature.

In another aspect, in general, the invention features displaying asecond object on the display device, so that after changing the positionof the first localized feature, the displayed position of the secondobject on the display device does not change in response to the changein the position of the first localized feature.

In another aspect, in general, the invention features determining thesecond position so as to make the position of the first object appear tocoincide with the position of the first localized feature as seen orfelt by the user.

In another aspect, in general, the invention features changing theorientation of the first coordinate reference frame in response to asignal being received by the computer.

In another aspect, in general, the invention features changing theorientation of the first coordinate reference frame in response to achange in the position of the first localized feature.

In another aspect, in general, the invention features changing theorientation of the first coordinate reference frame in response to asignal representative of the location of the user.

In another aspect, in general, the invention features changing theorientation of the first coordinate reference frame in response to asignal representative of a destination.

In another aspect, in general, the invention features changing theorientation of the first coordinate reference frame in response to asignal representative of a change in the user's immediate surroundings.

In another aspect, in general, the invention features changing theorientation of the first coordinate reference frame is changed inresponse to a signal representative of a change in the physiologicalstate or physical state of the user.

In another aspect, in general, the invention features redisplaying thefirst object further comprises changing the apparent size of the firstobject according to the change in position of the first localizedfeature.

In another aspect, in general, the invention features mounting aportable beacon, transponder or passive marker at a fixed point in theenvironment and determining the position vector of a second localizedfeature associated with the user's head relative to the fixed point.

In another aspect, in general, the invention features determining theposition vector of the first localized feature relative to the fixedpoint.

In another aspect, in general, the invention features mounting asourceless orientation tracker on a second user's head and determiningthe position of a localized feature associated with the body of thesecond user relative to the fixed point.

In another aspect, in general, the invention features determining theposition vector of a second localized feature associated with the user'shead relative to the fixed point without determining the distancebetween the second localized feature and more than one fixed point inthe environment.

In another aspect, in general, the invention features displaying thefirst object at a third position after displaying the first object atthe third position, changing the orientation of the display, and afterchanging the orientation of the display, continuing to display the firstobject at the third position.

In another aspect, in general, the invention features the first objectbeing a window in a wraparound computer interface.

In another aspect, in general, the invention features redisplaying thechanged position of the first localized feature not being within thefield of view of the display when the first object is redisplayed.

In another aspect, in general, the invention features displaying thefirst object at a position coinciding with the position of the firstlocalized object when the first localized object is within the field ofview of the display.

In another aspect, in general, the invention features positioning thefirst localized feature at a first point positioning the first localizedfeature at a second point and calculating the distance between the firstpoint and the second point.

In another aspect, in general, the invention features determining aposition vector of the first localized feature relative to a secondlocalized feature associated with the user's head and modifying theposition vector based on an orientation of the user's head.

In another aspect, in general, the invention features setting an assumedposition for the user's head in a coordinate system and setting aposition for the first localized feature in the coordinate system basedon the assumed position of the user's head and said position vector.

In another aspect, in general, the invention features measuring theorientation of the user's head relative to a fixed frame of reference.

In another aspect, in general, the invention features setting a virtualtravel speed and direction for the user modifying the assumed positionfor the user's head based on the user's virtual travel speed anddirection.

In another aspect, in general, the invention features mounting on thehead of a user a three degree of freedom orientation tracker fortracking the orientation of the head, and a three degree of freedomposition tracker for tracking the position of a first localized featureon the user's limb relative to a second localized feature on the user'shead, computing a position vector for the first localized featurerelative to the second localized feature, determining a rotation matrixbased on information received from the rotation tracker, andtransforming the position vector into a position vector for a fixedframe of reference based on the rotation matrix.

In another aspect, in general, the invention features using an acousticor radio frequency position tracker to track a position of a firstlocalized feature associated with a limb of the user relative to theuser's head.

In another aspect, in general, the invention features mounting a videocamera on the back of the user's head and displaying an image generatedby the video camera in a portion of a display device mounted on theuser's head.

In another aspect, in general, the invention features mounting a firstinertial sensor on a user's head, mounting a second inertial sensorelsewhere on the user's body or in an object held by the user, andtracking the position of one inertial sensor relative to the other.

Some embodiments of the invention include sensing data at the first andsecond inertial sensors and using the sensed data to track the positionof one inertial sensor relative to the other, tracking the position ofthe inertial sensor is done without reference to any signal receivedfrom a source not mounted on or held by the user and correcting thedrift of the relative position or orientation of the second inertialsensor relative to the first inertial sensor by measurements betweendevices on the user's head and devices elsewhere on the users body.

Among the advantages of the invention are one or more of the following.The device is easy to don, can track both head and hand, adds no newcables to a wearable computer system, works anywhere indoors or outdoorswith no preparation, and is simpler than alternatives such asvision-based self-tracking.

The details of one or more embodiments of the invention are set forth inthe accompanying drawings and the description below. Other features,objects, and advantages of the invention will be apparent from thedescription and drawings, and from the claims.

DESCRIPTION OF DRAWINGS

FIG. 1 is a perspective view of a self-referenced tracking devicemounted on a head.

FIG. 2 is a block diagram.

FIG. 3 is a graph of tracking coverage and relative resolution.

FIG. 4 is a view of an information cockpit.

FIG. 5 shows a user using a virtual reality game.

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

As seen in FIG. 1, implementations of the invention may combine asourceless head orientation tracker 30 with a head-worn tracking device12 that tracks a hand-mounted 3D beacon 14 relative to the head 16. Oneimplementation uses a wireless ultrasonic tracker 12, which has thepotential for low cost, lightweight, low power, good resolution, andhigh update rates when tracking at the relatively close ranges typicalof head-hand displacements.

As FIG. 1 illustrates, this arrangement provides a simple and easy todon hardware system. In a fully integrated wearable VR system using thistracker there are only three parts (a wearable computer 10, a headset 15with an integrated tracking system, and a hand-mounted beacon 14) andone cable connection 18. This is possible because the entire ultrasonicreceiver system 12 for tracking the beacon can be reduced to a few smallsignal-conditioning circuits and integrated with the sourcelessorientation tracker 30 in the head-worn display 15. By sharing themicroprocessor and its power and communications link to the wearable,the cost and complexity are reduced.

The benefits of this combination of elements stem from theserealizations:

1. It is usually not important to track the hand unless it is in frontof the head. Thus range and line-of-sight limitations are no problem ifthe tracker is mounted on the forehead.

2. The hand position measured in head space can be transformed intoworld space with good seen/felt position match using an assumed headpose, no matter how inaccurate.

3. Using one fixed beacon, the same tracking hardware can provide full6-DOF tracking.

Implementations of the invention may exhibit:

1. A new tracking concept that enables immersive visualization andintuitive manual interaction using a wearable system in arbitraryunprepared environments.

2. An information cockpit metaphor for a wearable computer userinterface and a set of interaction techniques based on this metaphor.

As shown in FIG. 2, a simple proof-of-concept implementation combines anInterSense IS-300 sourceless inertial orientation tracker 40 (availablefrom InterSense, Inc., in Burlington, Mass.) with a Pegasus FreeDultrasonic position tracker 50 (available from Pegasus Technologies Ltd.in Holon, Israel). The IS-300 has an “InertiaCube” inertial sensorassembly 42, just over an inch on a side, cabled to a smallcomputational unit 44 that outputs orientation data through a serialport 46. The FreeD product consists of a finger-worn wireless ultrasonicemitter 50A with two mouse buttons 54, and an L-shaped receiver bar 50Bwhich normally mounts on the frame of a computer monitor, and outputsx,y,z data through a serial port. For our experiments we mounted theInertiaCube and the L-shaped receiver bar on the visor 60 of a V-Cap1000 see-through HMD (available from Virtual Vision of Seattle, Wash.).The FreeD therefore measures the ring position relative to thehead-fixed coordinate frame whose orientation was measured by theIS-300.

Data from both trackers is transmitted to a PC 62 (Pentium 300 MHz,Windows 98) running a program 63 that uses Windows DirectX and Direct3Dcapabilities to display graphics and effect interaction techniques. Thegraphics output window of Direct3D is maximized to take control over theentire screen, and VGA output 64 (640×480 at 60 Hz) is passed into theV-Cap HMD as well as a desktop monitor.

The program 63 includes a tracker driver 71 and a fairly conventional VRrendering environment 72 that expects to receive 6-DOF head and handtracking data from the tracker driver as well as button states 65 forthe hand tracking device. The interaction techniques to be described areimplemented in the tracker driver. The basic functions of the trackerdriver, when tracking a single 3-DOF point on the hand, are:

1. Read in and parse the orientation data 68 from the IS-300 and theposition triad 70 from the FreeD.

2. Package the orientation data with the current head position inworld-frame, and output the combined 6-DOF data record 73 for the headto the VR program. The current assumed world-frame head position is thesame as the previous one unless the user is in the process of performinga navigation interaction such as flying. In this case the position isincremented based on the flying speed and direction.

3. Transform the hand position vector from head frame to world frame byfirst multiplying by the rotation matrix from head to world frameobtained from the orientation tracker, then adding the current assumedworld-frame head position. Output the result to the VR program as a3-DOF position record 74 for the hand device.

The simple implementation just described is wearable, but cannot beintegrated into an HMD elegantly, largely due to the size and powerconsumption of the IS-300 processing unit. A low-cost wearable versionusing available technologies could be implemented as follows:

The core of this implementation is an inertial head orientation modulecalled InterTrax 2 (available from InterSense and designed for use withconsumer HMDs such as the Sony Glasstron and Olympus EyeTrek). Usingtiny piezoelectric camcorder gyros, and solid-state accelerometers andmagnetometers, InterTrax 2 is designed as a single long narrow circuitboard 30 (FIG. 1) to lie across the top of the head mounted display unitalong the brow line. It is 9 cm long, 2 cm wide, and 0.5 cm thick withall components, except for a vertical gyro in the center, which sticksup 1 cm higher. It contains a low-power embedded 16-bit processor thatruns a simplified fixed-point version of the GEOS drift-correctedorientation-tracking algorithm used in the IS-300. It communicates tothe host through a single USB connector through which it draws itspower, and can be manufactured for very low cost in volume. It isexpected to achieve accuracy on the order of 2-3°, which is sufficientbecause the accuracy with which the hand avatar follows the physicalhand is totally independent of orientation tracking accuracy.

Another component is an embedded ultrasonic rangefinder (perhaps basedon the Pegasus FreeD technology). As shown in FIG. 1, three microphones80, 82, 84 and their ultrasonic pulse detection circuits together withthe InterTrax 2 board are embedded in a rigid plastic assembly designedto fit elegantly over the brow of an HMD. (In some embodiments, allcomponents would be embedded inside the HMD display unit while sharingthe HMD's cable 18, but in others, the added components are clipped on)The InterTrax 2 processor has enough unused timer inputs and processingbandwidth to timestamp the signals from the three ultrasonic pulsedetectors and relay this data down its USB link.

The ultrasonic tracking technology can be modified to take advantage ofthe very short range requirements. First, ultrasonic frequency may beincreased from 40 KHz to a higher frequency. This increases theattenuation in air, and virtually eliminates reverberation andinterference between nearby users. Second, the system can take advantageof the much reduced reverberation and the short time-of-flight toincrease the update rate of tracking to, say, 240 Hz, thus allowing thesystem to average 4 position samples for each 60 Hz graphics update, ortrack up to 4 beacons at 60 Hz. To calculate the resolution that thiswould yield in various parts of the tracking volume we calculated theGeometric Dilution of Precision (GDOP) throughout the tracking volumegiven the intended geometry of the microphone mounts on the headset. Theintended headset geometry, tracking range and optical field of view areillustrated superimposed on an isogram of a vertical slice through theGDOP data in FIG. 3. The plane of the microphones is angled downward 45°to insure that the system has tracking coverage for hands in the lap.The resolution at any point in space is the range measurement resolution(about 0.1 mm for short range ultrasonic measurements using 40 KHz)multiplied by the GDOP value, divided by 2 as a result of the 4×oversampling and averaging. Thus the expected resolution isapproximately 0.5 mm at a distance of 400 mm away from the headset.

A goal of a wearable computer is to keep the user's hands free toperform tasks. For this reason, the system uses a wireless 3-DOF ringpointer for interaction. The FreeD ring-mouse previously described isapproximately the right size. In some implementations of the system, thetracker will need to be triggered by a unique IR code from the headset,so that multiple beacons can be tracked.

In interactive visualization and design (IVD) and many other VRapplications, a pen-style input device may be more useful. Animplementation could use a wireless 5-DOF pen using the same basictechnology as the 3-DOF ring pointer, but employing two emitters thatare activated in an alternating sequence. A compact omni-directional pencould be implemented using cylindrical radiating ultrasonic transducersthat have been developed by Virtual Ink (Boston, Mass.), mounted at theends of a cylindrical electronics unit approximately the size of anormal pen, with two mouse buttons.

An additional device that could be included in the system and whoseapplications are discussed below is a small wireless anchor beacon thatcan be easily stuck to any surface. Ultrasonic beacons from InterSenseare of suitable size and functionality.

Portable VR Application

Object Selection and Manipulation Exploiting Proprioception

M. Mine, F. Brooks, and C. Sequin. (Moving Objects in Space: ExploitingProprioception in Virtual Environment Interaction. In SIGGRAPH 97Conference Proceedings, ACM Annual Conference Series, August, 1997),have discussed the benefits of designing virtual environment interactiontechniques that exploit our proprioceptive sense of the relative pose ofour head, hands and body. A variety of techniques were presented, suchas direct manipulation of objects within arms reach, scaled-world grab,hiding tools and menus on the users body, and body-relative gestures.

Implementations of the invention have advantages over conventionalworld-frame tracking systems for implementing these techniqueseffectively. With conventional trackers, any error in head orientationtracking will cause significant mismatch between the visualrepresentation of the virtual hand and the felt position of the realhand, making it difficult to accurately activate hidden menus while thevirtual hand is not in view. With implementations of the invention, thehead orientation accuracy is immaterial and visual-proprioceptive matchwill be good to the accuracy of the ultrasonic tracker—typically 1-2 mm.

Locomotion & View Control Tricks

This section describes a few techniques to permit user locomotion andview control.

Flying and Scaled-world Grab

The usual navigation interface device in fly-through virtualenvironments is a joystick. This is appropriate for a flight simulator,but reduces one's sense of presence in terrestrial environments, whereturning one's body toward the destination is more instinctive thanturning the world until the destination is in front. Implementations ofthe invention support this more immersive type of flying. No matter howone turns, if she raises a hand in front of her it will be trackable,and can be used to control flight speed and direction. Better yet, shecan use two-handed flying, which can be performed with the arms in arelaxed position and allows backwards motion, or the scaled-world grabmethod to reach out to a distant object and pull oneself to it in onemotion.

Walking Using Head Accelerometers as a Pedometer

For exploratory walk-throughs, the sense of presence is greatest forwalking, somewhat reduced for walking-in-place, and much further reducedfor flying. M. Slater, A. Steed and M. Usoh (The Virtual Treadmill: ANaturalistic Metaphor for Navigation in Immersive Virtual Environments.In First Eurographics Workshop on Virtual Reality, M. Goebel Ed. 1993),and M. Slater, M. Usoh and A. Steed (Steps and Ladders in VirtualReality. In Proc. Virtual Reality Software & Technology 94, G. Singh, S.K. Feiner, and D. Thalmann, Eds. Singapore: World Scientific, pages45-54, August 1994) have described a “virtual treadmill” technique inwhich a neural network is trained to recognize the bouncing pattern of aposition tracker on an HMD, and thus control virtual motion. Inertialhead-orientation trackers do not normally output the position obtainedby double integrating the accelerometers, because it drifts too much tobe useful, but it seems reasonable that pattern analysis of theacceleration signals would produce good results.

Head-Motion Parallax Using Anchor Beacon

When working with close objects, head motion parallax is an importantvisual cue. It can be achieved with the tracking system of the inventionon demand by using a trick. Normally, the system uses the 3-DOF positionvector from the user's head to the hand-mounted beacon to track theposition of the hand relative to the head, maintaining the head locationfixed. When desired, the user may hold the hand still (say on a desk),and push a button to reverse this process, so that the tracker driverinterprets the negative of the measured vector (in world frame) as aposition update of the head relative to the stationary hand. He can thenmove his head back and forth to look around an object, and release thebutton when his viewpoint is repositioned for optimal viewing. Afterflying or walking to an area, this may be a convenient way of makingfinely controlled viewpoint adjustments using natural neck motion. Notethat this operation is equivalent to grabbing the world and moving itaround with one's hand, which may be a more convenient maneuver whilestanding.

Implementations of the invention can perform full 6-DOF head trackingusing only one fixed reference point in the environment, while mostacoustic and optical trackers require at least three. This works in theinvention because head orientation is completely constrained by thesourceless head-tracker. This observation suggests another interestingtrick. One may carry an extra wireless anchor beacon in a pocket andplace it down on the table or stick it to a wall near a work area.Within range of this beacon, he can enjoy full 6-DOF tracking of bothhead and hand.

Wearable Computing Information Cockpit Interface

Information Cockpit Metaphor

In the field of wearable computing, three modes of displaying objects ina head-mounted display have been discussed. Head-stabilized objects aredisplayed at a fixed location on the HMD screen, so they move with yourhead motion and require no tracking. World-stabilized objects are fixedto locations in the physical environment. To cause them to stay fixeddespite user head-motion requires full 6-DOF head tracking.Body-stabilized objects are displayed at a fixed location on theinformation surround, a kind of cylindrical or spherical bubble ofinformation that follows the user's body position around. Headorientation tracking allows the user to look at different parts of thesurroundings by turning his head, but position tracking is not needed.

Pure head-stabilized displays are usually used with small opaquemonocular monitors mounted off to the side of the user's field of view.Without head tracking, this is better than having a display directly infront of the eye with information constantly blocking the frontal view.Use of this paradigm is widespread, and most of the wearable computervendors provide this style of untracked sidecar display. This is roughlyequivalent to wearing your desktop computer on your belt with themonitor mounted on a headband so that it is always available forhands-free viewing.

At the other end of the spectrum are world-stabilized AR displays, whichmust be implemented using see-through optics placed directly in front ofthe eyes. For a variety of applications such as surgery, constructionand maintenance, this is a highly valuable capability. However, itrequires sophisticated tracking and calibration, and is likely to remaina high-end subset of the total wearable computing market for quite a fewyears.

In the middle ground of complexity are the less common body-stabilizeddisplays, which also tend to be implemented with see through HMDs. Asimplemented by S. Feiner, B. MacIntyre, M. Haupt, and E. Solomon(Windows on the World: 2D Windows for 3D Augmented Reality. In Proc. ACMUIST 93. ACM Press, November 1993) objects were drawn on a 170°horizontal by 90° vertical portion of a sphere. To prevent userdisorientation, this hemispherical “virtual desk” was kept in front ofthe user's body by mounting an additional orientation tracker on theuser's torso, and using the difference between the head yaw and torsoyaw to pan the viewport. The desk was thus slaved to the user's torso,and the user could easily locate windows on it using his innateknowledge of head turn relative to the torso. This is intuitive but hasthe drawback that an additional orientation sensor must be mounted onthe user's torso. This adds cost, makes the system more difficult todon, and causes the virtual desk to shift around in response to slightpostural shifting of the user's torso, wobbling of the sensor mount, ormetallic distortion of the relative magnetic heading between the twosensors. An implementation of the invention uses a variation on thistheme, based on an “information cockpit” metaphor instead of abody-stabilized desk.

The information cockpit consists of a clear windshield, optionally drawnas a thin wireframe border, and a cluster of virtual instruments aroundit. As with the body-stabilized technique, the user's head is always inthe center of the cockpit, but the heading direction of the cockpitstays fixed until the user changes it. Generally, the user firstpositions the windshield towards the objects he will be working on withhis hands, and keeps the windshield area fairly clear of augmentationsso that he can see what he is doing. Thereafter, the user can turn tolook at the instruments, with or without turning his torso, and theinstruments will not move. To prevent the user from becoming disorientedor being forced to strain his neck as he moves around, theimplementation provides the user with steering techniques.

Outdoor Navigation Application

FIG. 4 shows an example of an information cockpit for an outdoornavigation application. The active field-of-view of the see-through HMDis indicated by heavy black rectangle 400. Thus only the augmentationswithin this rectangle are visible to the user, but rotating the headmoves this active view port around the scene and reveals the otheraugmentations once they are inside of it. In this example there are afew frequently-used icons 401 that are fixed (i.e. head-stabilized) inthe upper right of the heads-up display that will always be visible.There are additional icons 402 in the dashboard that are stabilized tothe information cockpit, and therefore can only be seen when the userlooks down a little to check them. Some of these are miniatureinformation instruments, such as dials and gauges, while others areicons used to bring up larger information instruments such as a webbrowser or interactive map display. By clicking on the map icon on thedashboard, the full-size map application window 404 pops up in themiddle of the active display area. The user may either quickly examineit then minimize it again, or save it for on-going reference by fixingit to a convenient spot on the information cockpit “windshield” 410 ashas been done in FIG. 4. The user can see a corner of the map in thecurrent view, but can look at the whole map again by looking up and tothe right. Virtual rear view mirrors 406 (fed by a video camera on theback of the head) have likewise been placed in three locations on thevirtual cockpit, but the user can re-position or close any of these fourinformation instruments at any time. In this example, the headingdirection of the cockpit is controlled by the application in order toguide the user to a destination. Using a GPS receiver in the user'swearable computer, the application orients the cockpit along thedirection from the user's current position to the destination, so heneed only follow the dotted lines 408 to their vanishing point on thehorizon to walk in the correct direction. This provides a virtualsidewalk in the forest, much as pilots are guided by virtualtunnel-in-the-sky displays. In an urban setting, the computer would usemap correlation to orient the cockpit along the current road in thesuggested walking direction.

Steering and Interaction

The ring tracker can be used for several purposes in wearable computerapplications: direct pointing to objects, virtual mouse pad cursorcontrol, command gestures, and measuring or digitizing.

Direct Pointing to Objects

When the ring tracker enters the viewing frustum of the HMD, the cursorjumps to the location of the ring and follows it. This provides rapiddirect selection of objects, taking full advantage of natural eye-handcoordination. In the virtual cockpit, one may glance up from thewindshield to a side panel, see an instrument he wants to use, reach outto exactly where he sees it and click on it with one of the ring buttonsto activate it or drag it into another view.

Many useful operations can be accomplished most easily with directselection and manipulation of objects. You can move and resize windows(i.e. instruments) the usual 2D way by dragging their borders. However,you can also exploit the 3D tracking of the ring to simultaneously moveand resize an instrument. Simply grab the title bar and pull it towardyou to make it larger or away from you to make it smaller, whilesimultaneously positioning it. If you pull it in towards your head farenough, as if to attach it to your HMD, it will change colors,indicating that if you let go of it, it will remain as a head-stabilizedobject. This is effectively like grabbing an instrument off your cockpitpanel and attaching it to your Heads-Up-Display (HUD) so that it willalways be visible in the foreground no matter where you look. By pushingit away far enough it will convert back to a cockpit panel instrument.

One of the cockpit windows that can be manipulated in a similar manneris the windshield itself. Simply click on any clear area of the “glass”where there aren't any graphical objects you might accidentally select,then drag it left/right or up/down to rotate the whole cockpit in space.This is one way of “steering” the cockpit, which is particularly usefulfor small course corrections or size adjustments or to refocus yourattention on another area of the workbench nearby.

Virtual Mouse Pad Cursor Control

Though fast and intuitive, the direct pointing technique would becomevery tiring if used to work with an instrument that requires extendedrepetitive clicking, such as a web browser or hypertext manual. Avirtual mouse pad technique can overcome this problem. As soon as theuser's hand drops below the viewing frustum of the HMD, the cursorcontrol automatically switches into this mode, in which left-and-rightmotion of the ring moves the cursor left-and-right, in-and-out motionmoves it up and down, and vertical position has no effect. This allowsthe user to rest his hand comfortably in his lap or on a desk, andcontrol the cursor by sliding his hand horizontally a few inches as ifon an imaginary mouse pad.

It is desirable that if the user positions the cursor on a particularobject then moves his head without moving the ring, the cursor willremain on the object. This means that the cursor is drawn as an objectin the cockpit-stabilized coordinates rather than the head-stabilizedscreen coordinates. This has several implications. First, the cursor isassociated with a point on the spherical information cockpit surface,only a portion of which is visible in the HMD, so the cursor could beout of view and quite difficult to find. A wiggling gesture is then usedto bring it back into the current center of display. Second, the ringtracking must be calculated in the cockpit stabilized coordinate frame,which means that if the user turns to the right, an “in-and-out” motionswitches from cockpit x-axis to y-axis and has an unexpected effect. Toavoid this, the ring position is transformed into cylindrical polarcoordinates and the radial and tangential components are used to controlcursor vertical and horizontal motion respectively.

Command Gestures

Ring tracker gestures may be used as a substitute for voice commands insituations where visual theatrics are more acceptable than audible ones,or where it is too noisy for reliable speech recognition. In general,gestures should commence outside of the direct pointing and virtualmouse pad regions, in order to avoid accidentally selecting and movingobjects. This leaves the sides and top of the viewing frustum, and thefirst few inches in front of the face (which are not used for directpointing). The gestures are executed by depressing a mouse button,possibly making a certain movement, then releasing the button. They arealways relative to the head in order to exploit proprioception, and thefact that the head is tracked, while the rest of the body is not. Manygestures may be defined, but the most commonly needed is a boresightcommand to reset the heading direction of the cockpit to the currentforward direction of the person's head as he walks about.

Measuring or Digitizing

Most people can hold their head very still, which opens the possibilitythat the ring tracker can be used to make measurements between twopoints that are close enough that both can be seen without moving thehead. This might be useful in an application such as taking inventory ofhow many pieces of each size are in a stockroom. Likewise, anapplication might ask you to quickly digitize a few corners of acomponent so it can determine based on the dimensions what model of thecomponent you are looking at and locate the appropriate manual pages.

To measure the distance between two close objects that are both withinthe display FOV at the same time, the user clicks both objects whileholding his head still. The distance is computed as the norm of thedifference of the two vector positions thus stored.

For two objects that are too far apart to be in the display FOV at once,a more elaborate procedure may be employed. The user first looks at thefirst object, positions the pointer beacon on it and depresses a button.At the moment the button is pressed, a world frame position vector (p1)of the first object is stored and then the tracking mode is switched to6-DOF tracking of the head relative to the stationary hand-held pointer,as previously described. While holding the pointer stationary on theobject and keeping the button depressed, the user then repositions hishead until the second object is in view, releases the button, and holdshis head still while moving the pointer to the second object, thenclicking it to capture the second position vector (p2) in the same worldcoordinate frame as the first. This technique may be practiced eitherwith a single pointing beacon operated by one hand, or using separatepointing beacons in each hand, to achieve approximately the samefunctionality as a conventional tape measure, but with the added benefitthat the measurements are automatically stored on a digital computer.

Relationships among remote objects may also be measured using standardtriangulation surveying methods, exploiting the functional similarity ofa see-through HMD optic with orientation tracker to a surveyor'stheodolite (although a tripod mounted theodolite is likely to be moreaccurate).

Mixed Display and AR-on-Demand Applications

The previous section presented the information cockpit as a specificvariation on Feiner's body-stabilized information surround. However, thecockpit metaphor also allows the user to make use of the head-stabilizedand world-stabilized coordinate frames at the same time. The previoussection gave one example of this in which the pilot drags informationfrom the cockpit onto the HUD, which makes it head-stabilized. Forexample, one may wish to have an alerting device always visible in theHUD that pops up notifications whenever a phone call, page or email isreceived, or when a scheduled meeting is about to begin, etc.

Likewise, one may wish to grab a certain instrument and paste it onto aphysical object in worldspace. For example, while debugging a circuitboard, you could overlay an interactive block diagram or schematic onthe board, and attach a virtual scope trace to your hand that is holdingthe scope probe (which is possible because the hand is tracked by thering pointer). To do this, you must first plant an anchor beacon, thenclick three corners of the circuit board to align the block diagram toit.

One important reason to plant anchor beacons is to create a shared ARworkspace for communication or collaboration with coworkers as describedin M. Billinghurst, S. Weghorst and T. Furness (Shared Space: AnAugmented Reality Approach for Computer Supported Cooperative Work.Virtual Reality Vol. 3(1) 1998) and D. Schmalstieg, A. Fuhrmann, Z.Szalavari, and M. Gervautz (Studierstube: An Environment forCollaboration in Augmented Reality. In CVE 96 Workshop Proceedings,September, 1996) incorporated by reference. Imagine a paperlessconstruction site with numerous workers building a structure accordingto the plans they are viewing on their wearable computers. It is nicethat they don't have to drag large rolls of blueprints around, but theyhave no way to stand around a blueprint and point to things. Thesolution is for someone to drop two anchor pins on a table, defining thetop two corners of a virtual blueprint or model that each person can seein correct perspective from his own vantage point.

A Variant Technique for Tracking the User's Hand

Some implementations of the invention use an inertial orientation sensorto track the rotation of the head, and an acoustic or optical positiontracker to track the position of the hand relative to the head. For manyapplications, the performance of the acoustic or optical positiontracker is sufficient. Furthermore, it has the great advantage that theitem being tracked can be a small wireless transponder, or even apassive marker. For some applications, such as the ring-mounted pointingdevice for wearable computing, this is an overwhelming advantage.

However, for some applications, such as a virtual reality game, it maybe desired to have the virtual object controlled by the hand tracker(e.g. a virtual sword or gun or racquet) respond to the hand motion withextremely fast smooth response. Acoustic, magnetic, or videometric handtrackers may introduce noticeable latency or jitter in theseapplications. Inertial position and orientation trackers are well knownto provide extremely low latency and low jitter, but they require driftcorrection, especially if tracking position and not just orientation isdesired. In a typical virtual reality application, the user's head andhand may both be tracked with 6 degrees of freedom relative to anexternal reference frame by using inertial sensors on the head and onthe hand to measure their motion with a high update rate and lowlatency. The drift of these inertial sensors is corrected by makingmeasurements with an ultrasonic, optical or magnetic tracking referencedevice mounted in the environment.

In some implementations of the present invention, the drift and latencyissues can be addressed without the requirement of a reference devicemounted in the environment. Foxlin, “Head-tracking Relative to a MovingVehicle or Simulator Platform Using Differential Inertial Sensors,”Proceedings of Helmet and Head-Mounted Displays V, SPIE vol. 4021 (2000)and co-pending U.S. patent application Ser. No. 09/556,135, which areincorporated herein by reference, describe techniques which enable theuse of inertial sensors to track the motion of an object relative to areference frame that is moving, even where the motion is not knowncompletely. These techniques require that inertial sensors be attachedto the moving body being used as the reference frame (in the citedreferences an example of the reference frame is given of a vehicle ormotion-platform and an example of the tracked object is given as a head;in the present invention, the moving reference frame may be the user'shead and the tracked object may be the user's hand or hand-mounted orhand-held object), as well as to the object being tracked (here, e.g.,the user's hand). The techniques utilize angular rate and linearacceleration signals from the sourceless orientation trackers on thereference frame and on the tracked object to derive a differentialinertial signal representative of the motion of the object relative tothe frame. In embodiments of the present invention, this technique maybe used to derive a differential inertial signal representative of themotion of the hand relative to the head.

FIG. 5 illustrates a user wearing a portable VR tennis game or trainingsystem. The computer and batteries are contained in backpack 502, whichis cabled to HMD 500 to which are mounted inertial sensors 506 andultrasonic transducers 510. He is holding a hand-held object 516, inthis case a tennis racquet, to which are attached inertial sensors 508and ultrasonic transducers 512. These hand-mounted devices may bepowered by their own batteries and communicate by wireless means to thesystem on the users head and torso, or there may be an additional cablebetween the racquet and the backpack. The signals from inertial sensors506 are processed by a first algorithm, preferably a drift-correctedinertial orientation tracking algorithm such as described in U.S. Pat.No. 5,645,077 to obtain a sourceless measurement of the headorientation. In addition, the signals from the hand-mounted inertialsensors 508 and the head-mounted inertial sensors 506 are jointlyprocessed to track both the position and orientation of the handrelative to the head, preferably using an algorithm such as described inFoxlin (2000) and co-pending U.S. patent application Ser. No.09/556,135. The drift of this relative inertial tracking is corrected bythe relative range measurements 514. In the illustrated system there arealso earphones 504 to provide 3D spatialized audio, and a hapticfeedback device 518 to provide tactile feedback to the user when thevirtual ball has hit the virtual racquet.

In general such a system may be used for other types of activities, suchas a sword-fighting or gun-fighting game or trainer, a surgical trainer,an immersive design environment, a human-computer interface, or anyother application known or not yet known which requires tracking of auser's head and one or more limbs or limb-mounted devices. While it isespecially advantageous for mobile or portable applications in which thecomputer is wearable, this is not a requirement, and the user may becabled to an off-body computer or communicate with an off-body computerthrough a wireless connection. In this case, it is still an advantage ofthe current invention that the tracking is accomplished without settingup an off-body reference device.

Other embodiments are within the scope of the claims.

The implementations described above track the hand with a head-mountedacoustic tracking system because this technology can be totally embeddedin a lightweight headset and achieve high resolution tracking over avery wide FOV.

However, the head mounted position tracker need not be acoustic. It maybe an electro-optical system which tracks LEDs, optical sensors, orreflective markers, or a video machine-vision device that recognizes thehands or fingers or some special markers mounted on the hands or fingersor handheld object, or even a magnetic tracker with a magnetic sourceheld in the hand and sensors integrated in the headset or vice versa, oran RF position locating device.

The implementation described above use inertial sourceless orientationtrackers. Other implementations may use other forms of head orientationtrackers, including trackers based on tilt-sensing or magnetic compasssensors, or any other form of head orientation tracker. In fact, someimplementations may use no head orientation tracker. In this case, thetracking system would not enable the user to look around in a virtualenvironment by turning his head, but it would still be useful for manualinteraction with computers using head-worn displays.

A number of embodiments of the invention have been described.Nevertheless, it will be understood that various modifications may bemade without departing from the spirit and scope of the invention.

What is claimed is:
 1. A method comprising mounting a sourcelessorientation tracker on a user's head, and using a position tracker totrack a position of a first localized feature associated with a limb ofthe user relative to the user's head.
 2. The method of claim 1 in whichthe first localized feature associated with the limb comprises a pointon a hand-held object or a point on a hand-mounted object or a point ona hand.
 3. The method of claim 2, wherein the first localized feature ison a stylus-shaped device.
 4. The method of claim 2, wherein the firstlocalized feature is on a ring.
 5. The method of claim 1 furthercomprising using the position tracker to determine a distance betweenthe first localized feature and a second localized feature associatedwith the user's head.
 6. The method of claim 1 in which the positiontracker comprises an acoustic position tracker.
 7. The method of claim 1in which the position tracker comprises an electro-optical system thattracks LEDs, optical sensors or reflective marks.
 8. The method of claim1 in which the position tracker comprises a video machine-vision devicethat recognizes the first localized feature.
 9. The method of claim 1 inwhich the position tracker comprises a magnetic tracker with a magneticsource held in the hand and sensors integrated in the headset or viceversa.
 10. The method of claim 1 in which the position tracker comprisesa radio frequency position locating device.
 11. The method of claim 1 inwhich the sourceless orientation tracker comprises an inertial sensor.12. The method of claim 1 in which the sourceless orientation trackercomprises a tilt-sensor.
 13. The method of claim 1 in which thesourceless orientation tracker comprises a magnetic compass sensor. 14.The method of claim 1 further comprising: mounting a display device onthe user's head; and displaying a first object at a first position onthe display device.
 15. The method of claim 14 further comprising:changing the orientation of the display device; and after changing theorientation of the display device, redisplaying the first object at asecond position on the display device based on the change inorientation.
 16. The method of claim 15, wherein the second position isdetermined so as to make the position of the first object appear to befixed relative to a first coordinate reference frame, which frame doesnot rotate with the display device during said changing of theorientation of the display device.
 17. The method of claim 16, whereinthe first object is displayed in response to a signal from a computer.18. The method of claim 17, further comprising: mounting a wearablecomputer on the user's body, and wherein the first object is displayedin response to a signal from the wearable computer.
 19. The method ofclaim 15, further comprising displaying a portion of a virtualenvironment on the display device.
 20. The method of claim 19, furthercomprising: displaying a portion of the virtual environment on thedisplay device before changing the orientation of the display device,and displaying a different portion of the virtual environment on thedisplay device after changing the orientation of the display device. 21.The method of claim 19, in which the virtual environment is afly-through virtual environment.
 22. The method of claim 19, in whichthe virtual environment includes a virtual treadmill.
 23. The method ofclaim 15, further comprising displaying a graphical user interface for acomputer on the display device.
 24. The method of claim 23, wherein thefirst object is a window, icon or menu in the graphical user interface.25. The method of claim 23, wherein the first object is a pointer forthe graphical user interface.
 26. The method of claim 16, furthercomprising: changing the position of the first localized featurerelative to the position tracker; and after changing the position of thefirst localized feature, redisplaying the first object at a secondposition on the display device determined based on the change in theposition of the first localized feature.
 27. The method of claim 26,further comprising: displaying a second object on the display device,wherein after changing the position of the first localized feature, thedisplayed position of the second object on the display device does notchange in response to the change in the position of the first localizedfeature.
 28. The method of claim 26, wherein the second position isdetermined so as to make the position of the first object appear tocoincide with the position of the first localized feature as seen orfelt by the user.
 29. The method of claim 17, further comprising:changing the orientation of the first coordinate reference frame inresponse to a signal being received by the computer.
 30. The method ofclaim 29, wherein the orientation of the first coordinate referenceframe is changed in response to a change in the position of the firstlocalized feature.
 31. The method of claim 29, wherein the orientationof the first coordinate reference frame is changed in response to asignal representative of the location of the user.
 32. The method ofclaim 29, wherein the orientation of the first coordinate referenceframe is changed in response to a signal representative of adestination.
 33. The method of claim 29, wherein the orientation of thefirst coordinate reference frame is changed in response to a signalrepresentative of a change in the user's immediate surroundings.
 34. Themethod of claim 29, wherein the orientation of the first coordinatereference frame is changed in response to a signal representative of achange in the physiological state or physical state of the user.
 35. Themethod of claim 27, wherein redisplaying the first object furthercomprises changing the apparent size of the first object according tothe change in position of the first localized feature.
 36. The method ofclaim 1, further comprising: mounting a portable beacon, transponder orpassive marker at a fixed point in the environment; and determining theposition vector of a second localized feature associated with the user'shead relative to the fixed point.
 37. The method of claim 36, furthercomprising determining the position vector of the first localizedfeature relative to the fixed point.
 38. The method of claim 36, whereinthe position vector is determined without determining the distancebetween the second localized feature and more than one fixed point inthe environment.
 39. The method of claim 36, wherein the position vectoris determined without determining the distance between the secondlocalized feature and more than two fixed points in the environment. 40.The method of claim 36, further comprising: mounting a sourcelessorientation tracker on a second user's head; and determining theposition of a localized feature associated with the body of the seconduser relative to the fixed point.
 41. The method of claim 16, furthercomprising: displaying the first object at a third position; afterdisplaying the first object at the third position, changing theorientation of the display; and after changing the orientation of thedisplay, continuing to display the first object at the third position.42. The method of claim 27, wherein the first object is a window in awraparound computer interface.
 43. The method of claim 26, wherein saidchanged position of the first localized feature is not within the fieldof view of the display when the first object is redisplayed.
 44. Themethod of claim 43, further comprising: displaying the first object atan apparent position coinciding with the position of the first localizedobject when the first localized object is within the field of view ofthe display.
 45. The method of claim 1, further comprising: positioningthe first localized feature at a first point; positioning the firstlocalized feature at a second point; and calculating the distancebetween the first point and the second point.
 46. The method of claim 1,further comprising: determining a position vector of the first localizedfeature relative to a second localized feature associated with theuser's head; and transforming the position vector based on anorientation of the user's head.
 47. The method of claim 46, furthercomprising: setting an assumed position for the user's head in acoordinate system; and setting a position for the first localizedfeature in the coordinate system based on the assumed position of theuser's head and said position vector.
 48. The method of claim 47, wheresetting a position for the first localized feature further comprises:measuring the orientation of the user's head relative to a fixed frameof reference.
 49. The method of claim 47, further comprising: setting avirtual travel speed and direction for the user; and modifying theassumed position for the user's head based on the user's virtual travelspeed and direction.
 50. The method of claim 1, wherein the sourcelessorientation tracker comprises a first inertial sensor, and furthercomprising: mounting a second inertial sensor elsewhere on the user'sbody or in an object held by the user; and tracking the position of oneinertial sensor relative to the other.
 51. The method of claim 14,further comprising: mounting a video camera on the back of the user'shead; and displaying an image generated by the video camera in a portionof the display device.
 52. A method comprising: using acoustic or radiofrequency signals to track a position of a first localized featureassociated with a limb of the user relative to the user's head.
 53. Atracking system comprising: an acoustic or radio frequency positiontracker adapted for mounting on a user's head, said tracker beingadapted to track a position of a first localized feature associated witha limb of the user relative to the user's head.
 54. A tracking systemcomprising a sourceless orientation tracker for mounting on a user'shead, and a position tracker adapted to track a position of a firstlocalized feature associated with a limb of the user relative to theuser's head.
 55. A method comprising: mounting a motion tracker on auser's head; using a position tracker to track a position of a firstlocalized feature associated with a limb of the user relative to theuser's head; positioning the first localized feature at a first point;positioning the first localized feature at a second point; andcalculating the distance between the first point and the second point.56. A system comprising: mounting a first inertial sensor on a user'shead; mounting a second inertial sensor elsewhere on the user's body orin an object held by the user; and tracking the position of one inertialsensor relative to the other.
 57. The method of claim 56, furthercomprising: sensing data at the first and second inertial sensors andusing the sensed data to track the position of one inertial sensorrelative to the other.
 58. The method of claim 57, wherein tracking theposition of the inertial sensor is done without reference to any signalreceived from a source not mounted on or held by the user.
 59. Themethod of claim 58, wherein the drift of the relative position ororientation of the second inertial sensor relative to the first inertialsensor is corrected by measurements between devices on the user's headand devices elsewhere on the users body.